Article: Tuesday, 26 November 2024
Imagine it’s a rainy afternoon and you’re considering a ride home from your office. Opening your Lyft app, you see a ride offer for $14.00. Would you accept it?
Perhaps you might, but what if the price was $13.99? Despite the minimal difference, analysts at Lyft discovered through an analysis of over 600 million price offers during a seven-month period that travelers were disproportionately drawn to fares just below a whole dollar. They estimated that if Lyft had adopted a 99-cent price strategy in 2019, profits would have increased by about $160M.
Now, switch the scenario to subscribing to a Substack newsletter. Would a price of $13.99 make you more likely to subscribe than $14.00? Interestingly, Substack’s recent experiment found the opposite effect as Lyft: round-priced newsletters were more attractive than those priced at 99 cents.
These cases highlight two key points. First, categorical boundaries like pricing thresholds play significant psychological and financial roles for consumers and businesses. Second, these boundaries can have opposing effects in different contexts. While 99-ending prices attracted more customers for Lyft, round prices proved more appealing for Substack subscribers.
In the social sciences, this issue is known as “external validity”—the extent to which findings from one context reliably generalize to another. A critical factor determining external validity is whether the observed effect is influenced by specific background characteristics that vary between the original and new contexts. If the effect does not interact with background differences, previous findings and models can help predict outcomes in new contexts. However, if background characteristics matter and vary across contexts, relying solely on past data will fail to predict future outcomes.
KU Leuven
Vlerick Business School
Italy is learning this the hard way. In 2020, the Italian government launched the “superbonus” program, offering an unprecedented 110% reimbursement for homeowners undertaking energy-efficient renovations. This decision was based on extrapolations from previous schemes. The State General Accounting Office expected the costs to merely double compared to previous schemes that offered 50% to 65% tax credits, predicting manageable expenditures of around €35 billion over a 15-year period. However, it turned out this generalization was unwarranted. Four years after its introduction, the “superbonus” has already turned into a €160 billion financial debacle. A dramatic miscalculation with potentially dire fiscal consequences for Italy.
To avoid the trap of overgeneralisation, we offer decision-makers and data analysts three recommendations:
Before extrapolating from a previous context, it is crucial to thoroughly map out the similarities and differences between the contexts. Anchoring on the current decision context is essential for this assessment.
For example, the State General Accounting Office in Italy failed to evaluate whether the assumptions of the models built out of data from previous schemes were met in the current situation. This oversight ignored key differences crucial to homeowners’ decisions. A 110% tax credit is not just double of 55%. It is disproportionately appealing because it exceeds the total cost of renovations. This effectively eliminated any financial risk for homeowners, aligning their interests entirely with builders and inadvertently encouraging them to inflate costs, sometimes fraudulently. Additionally, the program allowed homeowners to transfer their tax credits to third parties, enabling renovations at no direct cost regardless of the homeowners’ liquidity or taxable income levels. These crucial differences rendered past data, though seemingly useful, largely irrelevant to the current context.
When considering whether and how to extrapolate from previous data, it is essential to understand why past results have been observed. Generalisability is fundamentally a theoretical question, involving the identification of necessary conditions for a result to be observed and the disabling conditions that might eliminate or reverse it. Historical data, whether correlational or from randomised trials, rarely provides complete information on these aspects. By theorising about the reasons behind observed results, decision-makers can identify the necessary conditions for those results.
For instance, consider the extensive dataset Walmart obtained in 2020. In collaboration with a team of behavioural scientists, Walmart pharmacies across the United States tested and ranked the effectiveness of 22 alternative text messages to encourage people to get vaccinated. To what extent can Walmart trust that the most effective messages for getting people to the pharmacy would work in different contexts, such as getting people to shop? After the study, the team theorised about—and to some extent empirically validated—the critical factors explaining the differential effectiveness of the alternative messages. They found that the most effective messages framed the target product as individual items waiting for customers to pick them up. This analysis positions Walmart better for generalising (or not) from this highly specific context (flu vaccination) to new ones.
Theorising is challenging. Key background variables may not be observable in the available data, and theories are inherently incomplete, potentially overlooking relevant variables that differ across contexts. Moreover, theories could be incorrect if they are based on unreliable data. For these reasons, decision-makers should often consider collecting new data.
In a recent study with a national wholesaler of construction supplies, researchers investigated the effectiveness of a predictive sales analytics tool that estimates churn probabilities for individual construction businesses. The team tested an intervention aimed at fostering realistic expectations about the tool’s performance to increase its adoption among salespeople. Contrary to past evidence suggesting this intervention would boost adoption, the team found that salespeople used the tool less effectively when informed via a disclaimer that it might overestimate the risk of churn for some customers. Past evidence on these disclaimers was not predictive of this specific situation, and new data was invaluable in uncovering this inconsistency.
Sometimes, data is cheaper to collect than one might think. A simple survey of how Italian homeowners’ intentions to renovate vary across different repayment rates could have indicated that past results were not generalisable and provided more contextualised knowledge to predict the expenses of the superbonus. Randomised controlled trials are also often easy and fast for digital companies, as demonstrated by Substack. Unfortunately, available data can often obscure the need for more contextualised understanding, more theorising, and more data collection.
Want to improve your organisational decision-making using newfound insights fuelled by A.I. and machine learning? Discover Leading with Decision-Driven Analytics, a three-day short course at RSM led by Dr Bart de Langhe and Dr Gabriele Paolacci.
Science Communication and Media Officer
Corporate Communications & PR Manager
Rotterdam School of Management, Erasmus University (RSM) is one of Europe’s top-ranked business schools. RSM provides ground-breaking research and education furthering excellence in all aspects of management and is based in the international port city of Rotterdam – a vital nexus of business, logistics and trade. RSM’s primary focus is on developing business leaders with international careers who can become a force for positive change by carrying their innovative mindset into a sustainable future. Our first-class range of bachelor, master, MBA, PhD and executive programmes encourage them to become to become critical, creative, caring and collaborative thinkers and doers.
